INSTRUCTION-TUNED MODELS

BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfer (2024.naacl-long)

Challenge:	Recent advances in mathematical reasoning typically rely on massive scale . yet, can strong reasoning capabilities be induced in small language models under extreme constraints?
Approach:	They train small language models with a single GPU for under 24 hours . they find that adapters unlock significant plasticity in standard instruction-tuned models .
Outcome:	The proposed model training on a single GPU (48GB) achieves 40% Pass@1 on AIME 24 (an 11.1% improvement over baseline) the model training results show that the adapter capacity and initialization are critical factors.

Copied to clipboard

Akari Asai, Sneha Kudugunta, Xinyan Yu, Terra Blevins, Hila Gonen, Machel Reid, Yulia Tsvetkov, Sebastian Ruder, Hannaneh Hajishirzi

Eliciting Instruction-tuned Code Language Models’ Capabilities to Utilize Auxiliary Function for Code Generation (2024.findings-emnlp)

Challenge:	Recent advances in few-shot generalization in natural language processing focus on English.
Approach:	They propose a benchmark that unifies 15 diverse tasks across 54 languages in a sequence-to-sequence format and provides a fixed set of few-shot examples and instructions.
Outcome:	The proposed framework unifies 15 diverse tasks across 54 languages in a sequence-to-sequence format and provides a fixed set of few-shot examples and instructions.

Copied to clipboard

Seonghyeon Lee, Suyeon Kim, Joonwon Jang, HeeJae Chon, Dongha Lee, Hwanjo Yu

Programming Languages in NLP

Challenge:	Using auxiliary functions to implement functions is important for instruction-tuned models because it reduces the implementation difficulty of a target function compared to implementing them from scratch.
Approach:	They propose several ways to provide auxiliary functions to the models by adding them to the query or providing a response prefix to incorporate the ability to utilize auxiliary function with the instruction following capability.
Outcome:	The proposed models outperform the recent powerful language models, gpt-4o, in the code generation task.

UNIWIZ: A Unified Large Language Model Orchestrated Wizard for Safe Knowledge Grounded Conversations (2024.findings-acl)

Copied to clipboard

Souvik Das, Rohini Srihari

FuxiTranyu: A Multilingual Large Language Model Trained with Balanced Data (2024.emnlp-industry)

Challenge:	Large Language Models (LLMs) have made significant progress in integrating safety and knowledge alignment, but excessive focus on safety alignment can lead to unintended hallucinations.
Approach:	They propose a "safety-priming" method to generate synthetic safety data and overcome safety bottlenecks.
Outcome:	The proposed framework generates synthetic safety data and overcomes safety bottlenecks.

Copied to clipboard

Haoran Sun, Renren Jin, Shaoyang Xu, Leiyu Pan, null Supryadi, Menglong Cui, Jiangcun Du, Yikun Lei, Lei Yang, Ling Shi, Juesi Xiao, Shaolin Zhu, Deyi Xiong

Measuring Sycophancy of Language Models in Multi-turn Dialogues (2025.findings-emnlp)

Challenge:	Large language models exhibit significant performance discrepancies between high- and low-resource languages.
Approach:	They present an open-source multilingual LLM with 8 billion parameters and a multilingual instruction dataset.
Outcome:	The proposed model achieves consistent multilingual representations across languages.

Copied to clipboard

Jiseung Hong, Grace Byun, Seungone Kim, Kai Shu

Dialogue Systems & Conversational Agents

Challenge:	Prior research on sycophancy has focused on single-turn factual correctness, overlooking the dynamics of real-world interactions.
Approach:	They propose a new evaluation suite that assesses sycophantic behavior in multi-turn, free-form conversational settings.
Outcome:	The proposed evaluation suite measures how quickly a model conforms to the user and how frequently it shifts its stance under sustained user pressure.

From Language Modeling to Instruction Following: Understanding the Behavior Shift in LLMs after Instruction Tuning (2024.naacl-long)

Copied to clipboard

Xuansheng Wu, Wenlin Yao, Jianshu Chen, Xiaoman Pan, Xiaoyang Wang, Ninghao Liu, Dong Yu

Long-Context Language Modeling with Parallel Context Encoding (2024.acl-long)

Challenge:	Large Language Models (LLMs) have achieved remarkable success in aligning with user intentions.
Approach:	They develop local and global explanation methods and a feed-forward-based method for input-output attribution to investigate the impact of instruction tuning on user intentions.
Outcome:	The proposed method compares explanations from pre-trained and instruction-tuned models . it empowers LLMs to recognize the instruction parts of user prompts, it encourages response generation .

Copied to clipboard

Howard Yen, Tianyu Gao, Danqi Chen

Demystifying optimized prompts in language models (2025.emnlp-main)

Challenge:	Existing long-context models degenerate with retrieved contexts.
Approach:	They propose a framework that can be applied to existing decoder-only LLMs for context expansion.
Outcome:	The proposed framework can be applied to any existing decoder-only LLMs for context expansion.

Copied to clipboard

Rimon Melamed, Lucas Hurley McCabe, H Howie Huang

Do Prompt-Based Models Really Understand the Meaning of Their Prompts? (2022.naacl-main)

Challenge:	Modern language models (LMs) are not robust to out-of-distribution inputs.
Approach:	They investigate the composition of machine generated (“optimized”) prompts and the mechanisms by which LMs parse and build predictions from them.
Outcome:	The proposed prompts are primarily composed of punctuation and noun tokens, which are more rare in the training data.

Copied to clipboard

Albert Webson, Ellie Pavlick

Unlocking Large Audio-Language Models for Interactive Language Learning (2026.findings-eacl)

Challenge:	Recent studies show that prompts help models to learn faster in the same way that humans learn faster when provided with task instructions expressed in natural language.
Approach:	They experiment with 30 prompts manually written for natural language inference (NLI) they find that models can learn just as fast with many irrelevant or pathologically misleading prompts .
Outcome:	The proposed model can learn as fast with irrelevant or pathologically misleading prompts as with instructively “good” prompts.

Copied to clipboard

Hongfu Liu, Zhouying Cui, Xiangming Gu, Ye Wang

Instruction-following Evaluation through Verbalizer Manipulation (2024.findings-naacl)

Challenge:	Computer-Assisted Pronunciation Training (CAPT) systems provide unintuitive feedback that lacks actionable guidance.
Approach:	They propose to use audio-language models to provide more user-friendly feedback for pronunciation training.
Outcome:	The proposed model outperforms baselines on mispronunciation detection and suggestion generation.

Copied to clipboard

Shiyang Li, Jun Yan, Hai Wang, Zheng Tang, Xiang Ren, Vijay Srinivasan, Hongxia Jin

Sentiment Analysis

Challenge:	Existing benchmarks focus on common instructions that align well with what the model learned during training, but proficiency in responding to these instructions does not necessarily imply strong ability in instruction following.
Approach:	They propose a new instruction-following evaluation protocol called verbalizer manipulation that instructs the model to verbalize the task label with words aligning with model priors to different extents.
Outcome:	The proposed protocol can be integrated with any classification benchmark to examine the model’s reliance on priors and its ability to override them to accurately follow the instructions.

Do Prompt Positions Really Matter? (2024.findings-naacl)

Copied to clipboard

Junyu Mao, Stuart E. Middleton, Mahesan Niranjan

CONTRANS: Weak-to-Strong Alignment Engineering via Concept Transplantation (2025.coling-main)

Challenge:	Prompt-based learning models have a high level of interest due to their ability to perform zero-shot and fewshot tasks.
Approach:	They conduct the most comprehensive analysis to date of prompt position for diverse natural language processing tasks.
Outcome:	The proposed model is more robust than previous models and is consistent even in instruction-tuned models.

Copied to clipboard

Weilong Dong, Xinwei Wu, Renren Jin, Shaoyang Xu, Deyi Xiong

Challenge:	Large language models behave consistently with human goals, values and intentions, but are computationally expensive.
Approach:	They propose a framework that enables weak-to-strong alignment transfer via concept transplantation.
Outcome:	The proposed framework surpasses instruction-tuned models in terms of truthfulness.

Fine-Tuning Large Language Models with Sequential Instructions (2025.naacl-long)

Copied to clipboard

Hanxu Hu, Simon Yu, Pinzhen Chen, Edoardo Ponti

Distilling Instruction-following Abilities of Large Language Models with Task-aware Curriculum Planning (2024.findings-emnlp)

Challenge:	Existing instruction-tuned models struggle to adhere to a query with multiple intentions, which impairs their performance when the completion of several tasks is demanded by a single command.
Approach:	They develop an automatic process that turns existing data into diverse and complex task chains and a new benchmark to evaluate a model’s ability to follow all the instructions in a sequence.
Outcome:	The proposed model can follow instructions better and deliver higher results in coding, maths, and open-ended generation.

Copied to clipboard

Yuanhao Yue, Chengyu Wang, Jun Huang, Peng Wang

LLMs for Extremely Low-Resource Finno-Ugric Languages (2025.findings-naacl)

Challenge:	Instruction tuning aims to align large language models (LLMs) with open-domain instructions and human-preferred responses.
Approach:	They propose a multi-round distillation framework that uses an oracle LLM to select instructions that are difficult for a student LLM.
Outcome:	The proposed framework outperforms large language models and user-tuned models on several widely recognized benchmarks and multiple student LLMs.

Copied to clipboard

Taido Purason, Hele-Andra Kuulmets, Mark Fishel

Exploring Graph Learning Tasks with Pure LLMs: A Comprehensive Benchmark and Investigation (2026.findings-acl)

Challenge:	Low-resource languages such as those in the Finno-Ugric family are underrepresented in large language models.
Approach:	They propose to develop large language models for extremely low-resource languages . they focus on Vro, Livonian, and Komi, which are underrepresented .
Outcome:	The proposed models cover almost the entire cycle of creation, from data collection to instruction tuning and evaluation.

Copied to clipboard

Yuxiang Wang, Xinnan Dai, Wenqi Fan, Yao Ma

Personalize Your LLM: Fake it then Align it (2025.findings-naacl)

Challenge:	Recent studies focus on performance benchmarks without fully comparing LLMs to graph learning models.
Approach:	They evaluate off-the-shelf and instruction-tuned graph learning models across a variety of scenarios.
Outcome:	The proposed models outperform traditional graph learning models in few-shot settings, the authors show . their models out perform models with instruction tuning, and they show excellent generalization and robustness.

Copied to clipboard

Yijing Zhang, Dyah Adila, Changho Shin, Frederic Sala

User-Assistant Bias in LLMs (2026.findings-acl)

Challenge:	Existing personalization methods require fine-tuning of large language models for each user, rendering them prohibitively expensive for widespread adoption.
Approach:	They propose a retrieval-based personalization approach that uses self-generated personal preference data and representation editing to enable quick and cost-effective personalization.
Outcome:	The proposed approach outperforms two personalization baselines by 40% on various tasks.

Copied to clipboard

Xu Pan, Jingxuan Fan, Zidi Xiong, Ely Hahami, Jorin Overwiening, Ziqian Xie

Achieving Stronger Generation via Simple Contrastive Tuning (2024.findings-emnlp)

Challenge:	Modern large language models are typically trained using structured role tags . asymmetries in training data associated with different role tags can potentially introduce inductive biases.
Approach:	They propose a task-agnostic benchmark to test user–assistant bias in large language models . they find human-preference alignment amplifies user bias, reasoning fine-tuning reduces it .
Outcome:	The proposed benchmark tests show that most instruction-tuned models exhibit strong user bias . human-preference alignment amplifies user bias, while reasoning fine-tuning reduces it.

Copied to clipboard

Zhimeng Wang, Pinzheng Wang, Juntao Li, Yibin Chen, Min Zhang

EVIT: Event-Oriented Instruction Tuning for Event Reasoning (2024.findings-acl)

Challenge:	Recent years have witnessed remarkable progress in large language models (LLMs).
Approach:	They propose a framework for contrastive decoding to enhance instruction-tuned models.
Outcome:	The proposed framework improves model performance without additional data or computational resources.

Copied to clipboard

Zhengwei Tao, Xiancai Chen, Zhi Jin, Xiaoying Bai, Haiyan Zhao, Yiwei Lou

HumanEval Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generation Task (2025.findings-acl)

Challenge:	Large language models (LLMs) have made significant advances in event reasoning . however, smaller instruction-tuned models do not consistently demonstrate exceptional proficiency .
Approach:	They propose an event-oriented instruction tuning technique to train a large language model . they propose a structure named event quadruple which contains the structure and semantics of events .
Outcome:	The proposed model achieves competitive performances on event reasoning tasks.

Copied to clipboard

Zhaojian Yu, Yilun Zhao, Arman Cohan, Xiao-Ping Zhang

Programming Languages in NLP

Challenge:	Existing benchmarks for code generation tasks are inadequate, but performance declines on self-invoking tasks.
Approach:	They propose a general recipe for generating more challenging versions of existing benchmarks . they propose to use instruction-tuned models to evaluate LLMs on self-invoking code generation tasks .
Outcome:	The proposed model improves on humanEval and MBPP but on self-invoking code generation tasks.

Benchmark Profiling: Mechanistic Diagnosis of LLM Benchmarks (2025.emnlp-main)

Copied to clipboard

Dongjun Kim, Gyuho Shim, Yongchan Chun, Minhyuk Kim, Chanjun Park, Heuiseok Lim

Challenge:	Large Language Models are often judged by their scores on standard benchmarks, yet such scores often overstate real capability since they mask the mix of skills a task actually demands.
Approach:	They propose a diagnostic framework that decomposes benchmark performance into ten cognitively grounded abilities and computes an Ability Impact Score (AIS) AIS quantifies how much each ability contributes to a model’s success on a given benchmark.
Outcome:	The proposed framework decomposes performance into ten cognitively grounded abilities and computes an Ability Impact Score (AIS) that quantifies how much each ability contributes to a model’s success on a given benchmark.

MeetingQA: Extractive Question-Answering on Meeting Transcripts (2023.acl-long)

Copied to clipboard

Archiki Prasad, Trung Bui, Seunghyun Yoon, Hanieh Deilamsalehy, Franck Dernoncourt, Mohit Bansal

Robustness in NLP

Challenge:	Meeting transcripts are a promising domain for natural language tasks . lack of annotated data impedes research on other important tasks in this domain .
Approach:	They propose an extractive QA dataset comprising questions asked by meeting participants and corresponding responses.
Outcome:	The proposed dataset extracts questions asked by meeting participants and corresponding responses from transcripts.

On the Adaptation of Unlimiformer for Decoder-Only Transformers (2024.lrec-main)

Copied to clipboard

Kian Ahrabian, Alon Benhaim, Barun Patra, Jay Pujara, Saksham Singhal, Xia Song

Towards Enhancing Coherence in Extractive Summarization: Dataset and Experiments with LLMs (2024.emnlp-main)

Challenge:	despite efforts in the community, most common models have a context length of 4k or less.
Approach:	They propose to adapt a vector-retrieval augmentation method to decoder-only transformers . they also expand the experimental setup on summarization to include a new task and an instruction-tuned model .
Outcome:	The proposed model performs on par with a model with 2x the context length.

Copied to clipboard

Mihir Parmar, Hanieh Deilamsalehy, Franck Dernoncourt, Seunghyun Yoon, Ryan Rossi, Trung Bui

Information Extraction & Text Mining

Challenge:	Existing methods for extractive summarization lack coherence, despite improvements . a human-annotated dataset is used to improve coherency of extractive summary .
Approach:	They propose to use human-annotated datasets to create coherent extractive summaries . they use supervised fine-tuning and natural language user feedback to enhance coherence .
Outcome:	The proposed dataset shows that LLMs can produce coherent summaries with human feedback.

Merging Triggers, Breaking Backdoors: Defensive Poisoning for Instruction-Tuned Language Models (2026.acl-long)

Copied to clipboard

San Kim, Gary Lee

Leveraging In-Context Learning for Political Bias Testing of LLMs (2025.acl-long)

Challenge:	Large Language Models (LLMs) are vulnerable to backdoor attacks, where adversaries poison a small subset of data to implant hidden behaviors.
Approach:	They propose a training pipeline that immunizes instruction-tuned LLMs against backdoor attacks.
Outcome:	The proposed defenses lower attack success rates while preserving instruction-following ability.

Copied to clipboard

Patrick Haller, Jannis Vamvas, Rico Sennrich, Lena Ann Jäger

Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoning (2026.acl-long)

Challenge:	Existing probing methods for evaluating LLMs with political questions have limited stability and are unreliable.
Approach:	They propose to use human survey data as in-context examples to query LLMs with political questions to evaluate their potential biases.
Outcome:	The proposed task improves the stability of question-based bias evaluation and may be used to compare instruction-tuned models to their base versions.

Copied to clipboard

Zelin Tan, Hejia Geng, Xiaohang Yu, Mulei Zhang, Guancheng Wan, Yifan Zhou, Qiang He, Xiangyuan Xue, Heng Zhou, Yutao Fan, Zhong-Zhi Li, Zaibin Zhang, Guibin Zhang, Chen Zhang, Zhenfei Yin, Philip Torr, Lei Bai

Reasoning

Challenge:	elucidating scaling laws for large language models (LLMs) during pre-training remains unexplored.
Approach:	They characterize how model scale, data, and compute interact during pre-training . they find that large models consistently demonstrate superior compute and data efficiency .
Outcome:	The proposed scaling laws offer practical guidance for scaling reasoning capabilities through reinforcement learning post-training.

Truth Knows No Language: Evaluating Truthfulness Beyond English (2025.acl-long)

Copied to clipboard

Blanca Calvo Figueras, Eneko Sagarzazu, Julen Etxaniz, Jeremy Barnes, Pablo Gamallo, Iria de-Dios-Flores, Rodrigo Agerri

Machine Translation

Challenge:	a new benchmark evaluates the truthfulness of large language models (LLMs) based on imitative falsehoods.
Approach:	They propose a professionally translated extension of the TruthfulQA benchmark . it evaluates truthfulness in Basque, Catalan, Galician, and Spanish .
Outcome:	The proposed extension of the TruthfulQA benchmark evaluates truthfulness in Basque, Catalan, Galician, and Spanish.

Only a Little to the Left: A Theory-grounded Measure of Political Bias in Large Language Models (2025.acl-long)

Copied to clipboard

Mats Faulborn, Indira Sen, Max Pellert, Andreas Spitz, David Garcia

Large Language Models Are Overconfident in Their Own Responses (2026.findings-acl)

Challenge:	Political biases in language models can affect performance in many applications . political biased models are often left-leaning, but are generally more left- leaning for instruction-tuned models .
Approach:	They propose to use the Political Compass Test to measure political bias in language models . they use survey-based evaluation tools to test prompts and classify their political stances .
Outcome:	The proposed model is based on the Political Compass Test, but is not scientifically valid.

Copied to clipboard

Mario Sanz-Guerrero, Manuel Mager, Katharina Von Der Wense

Business as Rulesual: A Benchmark and Framework for Business Rule Flow Modeling with LLMs (2026.acl-long)

Challenge:	Prior work has shown that instruction-tuned large language models are less well calibrated than their base pre-trained counterparts.
Approach:	They propose a simple inference-time strategy that frams the model’s answer as user input during confidence elicitation.
Outcome:	The proposed approach reduces overconfidence and improves calibration by up to 26% without retraining.

Copied to clipboard

Chen Yang, Ruping Xu, Ruizhe Li, Bin Cao, Jing Fan

Fast-Decoding Diffusion Language Models via Progress-Aware Confidence Schedules (2026.findings-acl)

Challenge:	Existing benchmarks for extracting structured procedural knowledge from unstructured business documents are limited by simplistic schemas and shallow logical dependencies.
Approach:	They propose a framework for extracting structured procedural knowledge from unstructured business documents . they propose BREX, a carefully curated benchmark comprising 409 real-world business documents and 2,855 expert-annotated rules .
Outcome:	The proposed framework outperforms standard prompts in rule extraction and execution.

Copied to clipboard

Amr Mohamed, Yang Zhang, Michalis Vazirgiannis, Guokan Shang